The flag. CUDA_GRAPH_ARGS=(--cuda-graph-max-bs-decode 544) (TP8, line 153) and (--cuda-graph-max-bs-decode 128) (TP4, line 163) are only reachable when DP_ATTENTION=true, which covers 3 of the 5 search-space rows in dsv4-fp4-b300-sglang-agentic-hicache (DEP4-none, DEP4-hicache, DEP8-none). A repo-wide search shows --cuda-graph-max-bs-decode exists nowhere else in InferenceX — not in any other SGLang launcher, and a git log -S pickaxe search shows this commit is the very first time the string has ever appeared in this repository's history. The strongest counter-evidence sits right next to it: benchmarks/single_node/fixed_seq_len/dsv4_fp4_b300_sglang.sh runs the same DeepSeek-V4 FP4 B300 MegaMoE DEP8 configuration (same --moe-a2a-backend megamoe, same SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK) and sizes the decode graph to the identical value, 544, but with the plain --cuda-graph-max-bs (line 113). This script's own non-DP fallback (line 191) also falls back to the plain flag. That is about as close to an apples-to-apples control as you get, and it points at a typo/transcription slip rather than an intentional, SGLang-supported decode-specific flag.\n\nWhy it matters. sglang.launch_server parses its CLI via argparse, which aborts the process with SystemExit(2) ("unrecognized arguments") on any unknown flag, before CUDA context creation or model load. If --cuda-graph-max-bs-decode isn't recognized by the pinned image (lmsysorg/sglang:nightly-dev-cu13-20260707-b4155233), every DP-attention job in this sweep — the exact B300 MegaMoE DEP recipe this PR is explicitly trying to preserve and rerun — fails immediately at startup. The PR's stated validation (bash -n, YAML parsing, load_config_files) only checks shell/YAML syntax; none of it constructs or exercises the actual sglang.launch_server argv, so this would slip through and only be caught once the GPU jobs actually run.\n\nAddressing the refutation. One verifier argued this is refuted by the PR's own evidence: the OOM analysis in the description says jobs from run 29367944489 reached CUDA warmup and died inside deep_gemm.fp8_paged_mqa_logits — which is downstream of argument parsing, so if the flag were invalid, those jobs would have died at parse time instead, before ever touching CUDA. That's a real observation, but it rests on an unverifiable assumption: that predecessor PR #2145's script used the exact same -decode spelling and that this PR's DEP4 lines are a byte-for-byte carryover. I could not find #2145 in this repository's git history (no remote is configured, and no prior commit ever contains "cuda-graph-max-bs-decode" per git log -S), and the PR description itself calls this a rebuild "on the latest main" of a PR whose diff had to be manually reconstructed (excluding B200 and changelog changes). A manual reconstruction is exactly the kind of step where a flag name could be mistyped even if the original #2145 script used the correct plain flag — we simply cannot confirm from what's in front of us that the run 29367944489 logs correspond to a config using the -decode spelling rather than the correct one. Given that ambiguity, and given that the one piece of concrete, in-repo, apples-to-apples evidence (the sibling fixed_seq_len recipe) uses the plain flag for the identical 544 value, I still think this needs the author to explicitly confirm the flag against the pinned SGLang image before merge rather than relying on an inference about an unrelated historical run.\n\nStep-by-step proof of the failure mode (if the flag is indeed invalid):\n1. A search-space entry with dp-attn: true, e.g. tp: 8, ep: 8, is scheduled — one of the 3 affected rows.\n2. The script enters the DP_ATTENTION=true branch (line ~119) and, since TP=8, sets CUDA_GRAPH_ARGS=(--cuda-graph-max-bs-decode 544) (line 153).\n3. SGLANG_CMD is assembled including "" (line ~229) and executed via "" >> "" 2>&1 &.\n4. sglang.launch_server's argparse.ArgumentParser.parse_args() encounters --cuda-graph-max-bs-decode, finds no matching add_argument, and calls parser.error(...), which prints "unrecognized arguments: --cuda-graph-max-bs-decode 544" and exits with status 2 — before any CUDA/model initialization.\n5. wait_for_server_ready polls the health port, never gets a response (the process already exited), and the job fails/hangs on the timeout instead of ever reaching the benchmark — for all 3 dp-attn rows, defeating the PR's stated purpose of rerunning exactly this B300 MegaMoE DEP configuration.\n\nFix. Confirm --cuda-graph-max-bs-decode is a recognized argument in the pinned SGLang nightly's server_args.py/launch_server before merge (e.g. via python -m sglang.launch_server --help against the actual image, or grepping the vendored SGLang source in the container). If it isn't, change both occurrences (lines 153, 163) to the plain --cuda-graph-max-bs, consistent with the sibling MegaMoE recipe and this script's own non-DP fallback. While touching this block, the comment directly above ("DEP8 (dp 8) serves conc up to 512") is also worth updating since this PR's own configs/nvidia-master.yaml diff removes 512 from the DEP8 conc-list, leaving 196 as the new max — a cosmetic follow-up, not a blocker on its own.

What the bug is: The PR's stated purpose is to "rerun only the B300 configuration" for dsv4-fp4-b300-sglang-agentic-hicache with a lowered mem-fraction-static (0.93 → 0.90) to fix an observed CUDA OOM, plus a broader recipe overhaul (MegaMoE DEP path, a new low-latency TP-only path, several new SGLANG_* env vars, and revised conc-list search-space entries). The changed-files set for this PR is exactly two files — benchmarks/single_node/agentic/dsv4_fp4_b300_sglang.sh and configs/nvidia-master.yaml. No entry is appended to perf-changelog.yaml.

The code path that triggers it: .github/workflows/run-sweep.yml gates both its push (lines 25-26) and pull_request (lines 35-36) triggers with paths: ["perf-changelog.yaml"]. Since this PR never touches that file, the entire sweep workflow (including check-changelog, the reuse-sweep gate, and the actual benchmark jobs) will not fire on this PR, and will not fire on the post-merge push to main either. Even setting aside the workflow-level path filter, the sweep's matrix is itself derived from perf-changelog.yaml: utils/process_changelog.py's get_added_lines() git-diffs only that file between base and head, and main() explicitly raises ValueError("No additions found in the changelog file.") when there are no appended lines (utils/process_changelog.py:128-129). The config-keys from each appended changelog entry are what get resolved against the master config via get_config_keys_from_master() to build the actual sweep — so an empty changelog diff means zero configs get selected, config-key resolution never runs, and the dsv4-fp4-b300-sglang-agentic-hicache search-space this PR just edited is never benchmarked.

Why existing code doesn't prevent it: There is no other trigger path. Labels like full-sweep-fail-fast only affect fail-fast behavior of matrix jobs that are already running — they don't bypass the perf-changelog.yaml path filter. validate_perf_changelog.py (or equivalent validation) does not error out when the changelog is left unchanged, so nothing blocks the merge; the sweep just silently never runs. Recent config-changing PRs in this repo (e.g. #2158/#2222, #2197, #2198) consistently append a changelog entry alongside their config edits, and the immediate predecessor to this PR (#2145) also lacks one, showing this omission isn't a one-off outlier but a real gap that AGENTS.md and the PR checklist both explicitly call out as required ("append a perf-changelog.yaml entry (required — triggers benchmarks)").

Impact: Concretely, merging this PR as-is makes it a functional no-op for its own stated purpose. The mem-fraction-static OOM fix, the MegaMoE DEP path, the new low-latency TP-only path, and the revised concurrency search-space all land on main, but none of them get exercised by CI — the exact scenario (DEP4 CUDA OOM in deep_gemm.fp8_paged_mqa_logits) that motivated this PR will not be re-validated. A maintainer or the author would need a follow-up PR just to actually trigger the benchmark.

Step-by-step proof:

1. git diff main...HEAD -- perf-changelog.yaml for this PR is empty (only dsv4_fp4_b300_sglang.sh and nvidia-master.yaml are in <changed-files>).
2. On pull_request, GitHub Actions evaluates run-sweep.yml's trigger: the event's changed-file set intersected with paths: ["perf-changelog.yaml"] is empty, so the workflow does not run at all, regardless of labels applied.
3. Suppose the workflow were manually forced to run anyway (e.g., via workflow_dispatch): the setup job calls process_changelog.py --base-ref ... --head-ref ..., which calls get_added_lines() — a git diff scoped to perf-changelog.yaml only. Since no lines were added there, added_yaml.strip() is empty and main() raises ValueError("No additions found in the changelog file."), hard-failing before any config-key resolution or sweep construction happens.
4. Either way, the dsv4-fp4-b300-sglang-agentic-hicache config — the one this PR just modified — never gets included in a benchmark run.

How to fix: Append a perf-changelog.yaml entry at the end of the file with config-key: dsv4-fp4-b300-sglang-agentic-hicache, a short description of the mem-fraction/MegaMoE/low-latency changes, and a link to this PR — mirroring the format used by the sibling PRs (#2158/#2222, #2197, #2198) that touched nvidia-master.yaml.